Accessing a NetCDF4/HDF5 File - S3 Direct Access

Summary

In this notebook, we will access monthly sea surface height from ECCO V4r4 (10.5067/ECG5D-SSH44). The data are provided as a time series of monthly netCDFs on a 0.5-degree latitude/longitude grid.

We will access a single netCDF file from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an xarray dataset. This approach leverages S3 native protocols for efficient access to the data.

Requirements

1. AWS instance running in us-west-2

NASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.

2. Earthdata Login

An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.

3. netrc File

You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: Authentication for NASA Earthdata.

Learning Objectives

how to retrieve temporary S3 credentials for in-region direct S3 bucket access
how to perform in-region direct access of ECCO_L4_SSH_05DEG_MONTHLY_V4R4 data in S3
how to plot the data

Import Packages

%matplotlib inline
import matplotlib.pyplot as plt
import os
import requests
import s3fs
from osgeo import gdal
import xarray as xr
import hvplot.xarray
import holoviews as hv

Get Temporary AWS Credentials

Direct S3 access is achieved by passing NASA supplied temporary credentials to AWS so we can interact with S3 objects from applicable Earthdata Cloud buckets. For now, each NASA DAAC has different AWS credentials endpoints. Below are some of the credential endpoints to various DAACs:

s3_cred_endpoint = {
    'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',
    'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',
    'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',
    'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',
    'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'
}

Create a function to make a request to an endpoint for temporary credentials. Remember, each DAAC has their own endpoint and credentials are not usable for cloud data from other DAACs.

def get_temp_creds(provider):
    return requests.get(s3_cred_endpoint[provider]).json()

temp_creds_req = get_temp_creds('podaac')
#temp_creds_req

Set up an `s3fs` session for Direct Access

s3fs sessions are used for authenticated access to s3 bucket and allows for typical file-system style operations. Below we create session by passing in the temporary credentials we recieved from our temporary credentials endpoint.

fs_s3 = s3fs.S3FileSystem(anon=False, 
                          key=temp_creds_req['accessKeyId'], 
                          secret=temp_creds_req['secretAccessKey'], 
                          token=temp_creds_req['sessionToken'])

In this example we’re interested in the ECCO data collection from NASA’s PO.DAAC in Earthdata Cloud. Below we specify the s3 URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs.

s3_url = 's3://podaac-ops-cumulus-protected/ECCO_L4_SSH_05DEG_MONTHLY_V4R4/SEA_SURFACE_HEIGHT_mon_mean_2015-01_ECCO_V4r4_latlon_0p50deg.nc'

Direct In-region Access

Open with the netCDF file using the s3fs package, then load the cloud asset into an xarray dataset.

s3_file_obj = fs_s3.open(s3_url, mode='rb')

ssh_ds = xr.open_dataset(s3_file_obj, engine='h5netcdf')
ssh_ds

Get the SSH variable as an xarray dataarray

ssh_da = ssh_ds.SSH
ssh_da

<xarray.DataArray 'SSH' (time: 1, latitude: 360, longitude: 720)>
[259200 values with dtype=float32]
Coordinates:
  * time       (time) datetime64[ns] 2015-01-16T12:00:00
  * latitude   (latitude) float32 -89.75 -89.25 -88.75 ... 88.75 89.25 89.75
  * longitude  (longitude) float32 -179.8 -179.2 -178.8 ... 178.8 179.2 179.8
Attributes:
    coverage_content_type:  modelResult
    long_name:              Dynamic sea surface height anomaly
    standard_name:          sea_surface_height_above_geoid
    units:                  m
    comment:                Dynamic sea surface height anomaly above the geoi...
    valid_min:              -1.8805772066116333
    valid_max:              1.4207719564437866

xarray.DataArray

'SSH'

time: 1
latitude: 360
longitude: 720

...
```
[259200 values with dtype=float32]
```
Coordinates: (3)
- time
  (time)
  datetime64[ns]
  2015-01-16T12:00:00
  axis :
  T
  bounds :
  time_bnds
  coverage_content_type :
  coordinate
  long_name :
  center time of averaging period
  standard_name :
  time
```
array(['2015-01-16T12:00:00.000000000'], dtype='datetime64[ns]')
```
- latitude
  (latitude)
  float32
  -89.75 -89.25 ... 89.25 89.75
  axis :
  Y
  bounds :
  latitude_bnds
  comment :
  uniform grid spacing from -89.75 to 89.75 by 0.5
  coverage_content_type :
  coordinate
  long_name :
  latitude at grid cell center
  standard_name :
  latitude
  units :
  degrees_north
```
array([-89.75, -89.25, -88.75, ...,  88.75,  89.25,  89.75], dtype=float32)
```
- longitude
  (longitude)
  float32
  -179.8 -179.2 ... 179.2 179.8
  axis :
  X
  bounds :
  longitude_bnds
  comment :
  uniform grid spacing from -179.75 to 179.75 by 0.5
  coverage_content_type :
  coordinate
  long_name :
  longitude at grid cell center
  standard_name :
  longitude
  units :
  degrees_east
```
array([-179.75, -179.25, -178.75, ...,  178.75,  179.25,  179.75],
      dtype=float32)
```
Attributes: (7)
coverage_content_type :
modelResult
long_name :
Dynamic sea surface height anomaly
standard_name :
sea_surface_height_above_geoid
units :
m
comment :
Dynamic sea surface height anomaly above the geoid, suitable for comparisons with altimetry sea surface height data products that apply the inverse barometer (IB) correction. Note: SSH is calculated by correcting model sea level anomaly ETAN for three effects: a) global mean steric sea level changes related to density changes in the Boussinesq volume-conserving model (Greatbatch correction, see sterGloH), b) the inverted barometer (IB) effect (see SSHIBC) and c) sea level displacement due to sea-ice and snow pressure loading (see sIceLoad). SSH can be compared with the similarly-named SSH variable in previous ECCO products that did not include atmospheric pressure loading (e.g., Version 4 Release 3). Use SSHNOIBC for comparisons with altimetry data products that do NOT apply the IB correction.
valid_min :
-1.8805772066116333
valid_max :
1.4207719564437866

Plot the SSH dataarray for time 2015-01-16T12:00:00 using hvplot.

ssh_da.hvplot.image(x='longitude', y='latitude', cmap='Spectral_r', aspect='equal').opts(clim=(ssh_da.attrs['valid_min'],ssh_da.attrs['valid_max']))

Resources

Direct access to ECCO data in S3 (from us-west-2)

Data_Access__Direct_S3_Access__PODAAC_ECCO_SSH using CMR-STAC API to retrieve S3 links